Seems like users shouldn't need to do this manually? Does this mean our internal logic is not working?

It's more that we've written a possibly strange example.
It's not using an ActionBar, instead, there's an ActionGroup (its own focus scope) that is a sibling of the ListBox. Moreover, it's only rendered when there is a selection. Once the items are removed, there's no more selection, so the ActionGroup disappears immediately, before focus is restored there, and the ActionGroup doesn't restore focus, so it was just lost to the body.

It wasn't really related to the problem being solved in this PR, so I haven't given this particular issue more thought, but it should definitely be easier.